Skip to content

adapter: don't leak arrangement compression into the realized config - #38112

Draft
aljoscha wants to merge 6 commits into
aljoscha/cluster-legacy-04-zdt-flagfrom
aljoscha/cluster-legacy-06-compression-fix
Draft

adapter: don't leak arrangement compression into the realized config#38112
aljoscha wants to merge 6 commits into
aljoscha/cluster-legacy-04-zdt-flagfrom
aljoscha/cluster-legacy-06-compression-fix

Conversation

@aljoscha

@aljoscha aljoscha commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Stacked on #38104.

Arrangement compression is a replica config shape dimension: it is part of ReconfigurationTarget, the reconcile kernel distinguishes replicas by it, and the controller transitions it through the hydrate-overlap like SIZE. Several hand-maintained dimension lists forgot it.

The reshape leak. reshape_alter_cluster_managed writes the reconfiguration record while resetting the config shape carried by new_config back to the realized values, deferring the transition to the record's cut-over. The reset enumerated size, replication factor, availability zones, and logging, but missed arrangement compression. A compression-carrying ALTER thus flipped the realized config immediately: the baseline strategy then desired the new shape and bounced the baseline replicas mid-reconfiguration, defeating the graceful overlap, and the cut-over later wrote the same value again. The reset now goes through realized_reconfiguration_target / apply_reconfiguration_target instead of hand-listing dimensions, so a target dimension can no longer be missed at this site.

The planner gate. The WAIT gate rejected a compression-only ALTER as having nothing to wait on, even though such an ALTER transitions through a hydrate-overlap like any other shape change. It is now allowed (and the error message names the option).

The observability surface. mz_cluster_reconfigurations.changes diffed only the other four dimensions, so a compression-only record reported an empty diff while in progress, and the SHOW CLUSTERS activity summary showed such a cluster as steady mid-reconfiguration. Both now report the dimension.

Tests. The compression ALTERs in managed_cluster.slt relied on the leak: their SHOW CREATE readback only saw the new value because the reshape wrote it early. They now request the synchronous cut-over (WAIT FOR '0s'), which advances the realized config in the ALTER's own transaction. ALTER ... RESET takes no WITH clause, so the RESET spelling is exercised for acceptance without a readback. cluster-controller.td gains regression coverage: the cc_preserve section asserts that a folding compression ALTER lands in the record's target and changes while the realized config stays untouched and only the overlap replica is bounced, and a new section runs a compression-only ALTER through the record path end to end.

Known gap, left for a follow-up: the audit event's reconfiguration details intentionally omit compression, and a shape-dimension RESET cannot carry a WAIT clause (the parser only accepts WITH on SET).

Motivation

Fixes a bug in a feature that has yet to be released.

Checklist

  • This PR has adequate test coverage / QA involvement has been duly considered. (trigger-ci for additional test/nightly runs)
  • This PR has an associated up-to-date design doc, is a design doc (template), or is sufficiently small to not require a design.
  • If this PR evolves an existing $T ⇔ Proto$T mapping (possibly in a backwards-incompatible way), then it is tagged with a T-proto label.
  • If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label (example).
  • If this PR includes major user-facing behavior changes, I have pinged the relevant PM to schedule a changelog post.

…duler

The cluster controller has been default-on since v26.29 and owns the replica
set of every user managed cluster. The break-glass dyncfg kept two legacy
paths reachable: the REFRESH scheduler in `cluster_scheduling.rs`, whose two
entry points already returned before doing any work while the gate was on,
and the legacy branches in the ALTER sequencer.

Delete the gate and everything only it kept alive: the scheduler module, its
coordinator plumbing (two messages, the timer, the select-loop tick, the
`cluster_scheduling_decisions` state), the two scheduler metrics, the
`cluster_check_scheduling_policies_interval` system var, and the
`ReplicaCreateDropReason::ClusterScheduling` variant.

The persisted audit vocabulary stays: `SchedulingDecisionsWithReasonsV2` and
friends are written by the controller's on-refresh path too, and old events
must remain decodable.
`ManagedClusterIds` and the sequencer's two ownership tests excluded
system/builtin clusters, so the sequencer kept a second, complete
replica-materialization implementation alive just for them. The exception was
originally load-bearing (the boot-time builtin replica migration and the
controller would have been two conflicting writers), but
`reconcile_builtin_cluster_replicas` now converges a builtin cluster's replica
set on the cluster's own managed config, the same config the controller derives
its targets from, so the two converge by construction.

Drop the `is_user()` conjuncts. Runtime ALTERs of system clusters now flow
through the controller like any other managed cluster: a config-shape change
reshapes into a durable reconfiguration record, a factor change updates the
config and the controller converges the replica set within a tick.

The boot migration and the controller compose in both directions. The
controller matches replicas by shape and count, never by name, so the
migration-created `r1..rN` satisfy its baseline. The migration converges by
canonical name, so a boot after a reshape renames or re-creates replicas the
controller materialized under generator names, which is harmless churn on
processes that are cold at boot anyway.
With the controller owning every managed cluster's replica set,
`NeedsFinalization::Yes` has no producer. Delete the machine it drove: the
`WaitForHydrated` and `Finalize` stages, the `-pending` overlap replicas, the
`pending_cluster_alters` connection state and its retire paths, and the
`AlterClusterWhilePendingReplicas` error.

The direct reshape path is deliberately kept, as the synchronous cut-over. It
is now routed to by an explicitly zero-timeout commit strategy
(`WITH (WAIT FOR '0s')`, or `WAIT UNTIL READY (TIMEOUT '0s', ON TIMEOUT
'COMMIT')`) rather than by the absence of a `WAIT` clause. Two reasons. It is
the escape hatch: every other reshape depends on the controller ticking and
applying, and this is the one that still works when the controller itself is
the problem, while also unsticking a wedged reconfiguration by retiring its
record. And the semantics are honest: a zero timeout with commit already means
"cut over now, hydrated or not", so doing it synchronously in the ALTER is the
same outcome minus a tick.

"The same outcome" has to be true, so the cut-over does not improvise. It
folds its target onto an in-flight one exactly as the reshape path does, and it
converges the replica set with the controller's own reconcile kernel, so a
replica that already has the target shape is kept rather than bounced. Forcing
a stuck-but-hydrating resize to commit therefore keeps the replica that was
already up, and lands the record on `finalized` (forced) rather than
`cancelled`, since the cut-over reached the record's own target.

Creating a replica from the controller's `ReplicaShape` also drops a lossy
round-trip through the planner's `ComputeReplicaConfig`, which cannot represent
`INTROSPECTION DEBUGGING` without an interval. A cluster in that state used to
get a replica whose logging disagreed with the config that called for it.
Follow-ups on deleting the staged reconfiguration machine, no behavior
change.

Collapse the cut-over routing's bool-plus-Option pair into a single
optional cut-over target, computed unconditionally on that path. The fold
with nothing in flight returns exactly the statement's shape, so applying
it is the identity and the "nothing in flight, nothing to fold" special
case disappears. The cut-over now computes its target through the same
call the reshape path makes, so the two agreeing is structural rather
than commented.

Applying a target onto a managed config is the inverse of
`realized_reconfiguration_target`, so it moves next to it in objects.rs
as `apply_reconfiguration_target`, keeping the exhaustive-destructure
compile guard beside the type it guards.

`sequence_create_managed_cluster` builds the one `ReplicaShape` all its
replicas share once, instead of a separate logging binding plus a
per-iteration shape construction in the loop.

Also fix the `sequence_alter_cluster_managed_to_managed` doc, which
claimed the carried record is retired as cancelled and the replica set
dropped wholesale: `retire_carried_reconfiguration` settles an on-target
record as finalized (forced), and the reconcile keeps already-matching
replicas.
Graceful cluster reconfiguration has been behind the
`enable_zero_downtime_cluster_reconfiguration` feature flag, default off, so the
`WITH (WAIT ...)` surface was rejected at plan time unless a deployment turned
it on. That is now the only thing standing between an operator and the
synchronous cut-over, which is the escape hatch for a reshape when the cluster
controller itself is the problem. A break-glass path behind a default-off flag
is not a break-glass path.

Remove the flag and the planner gate. The two rejections that share that code
path stay: a `WAIT` without a replica-shape change, and a `WAIT` on an
unmanaged cluster.

Every test that used the surface enabled the flag itself, so those statements
go. In a mixed-version run some phases execute against a released binary that
still enforces the gate (platform-checks' graceful-reconfiguration check runs
its first manipulate phase there), so `get_minimal_system_parameters` pins the
flag on below v26.38 instead.

The docs drop the private-preview badges: the surface is generally available
once nothing gates it.
@aljoscha
aljoscha requested review from a team as code owners August 7, 2026 11:44
@aljoscha
aljoscha marked this pull request as draft August 7, 2026 11:49
@aljoscha
aljoscha force-pushed the aljoscha/cluster-legacy-06-compression-fix branch 2 times, most recently from 6ce5344 to cf73bc0 Compare August 7, 2026 12:28
Arrangement compression is a replica config shape dimension: it is part of
`ReconfigurationTarget`, the reconcile kernel distinguishes replicas by it,
and the controller transitions it through the hydrate-overlap like SIZE.
Several hand-maintained dimension lists forgot it.

The reshape path writes the reconfiguration record while resetting the
config shape carried by `new_config` back to the realized values, deferring
the transition to the record's cut-over. The reset enumerated size,
replication factor, availability zones, and logging, but missed arrangement
compression. A compression-carrying ALTER thus flipped the realized config
immediately: the baseline strategy then desired the new shape and bounced
the baseline replicas mid-reconfiguration, defeating the graceful overlap,
and the cut-over later wrote the same value again. Reset via
`realized_reconfiguration_target` and `apply_reconfiguration_target`
instead of hand-listing dimensions, so a target dimension can no longer be
missed here.

The planner's WAIT gate had the same gap: it rejected a `WAIT` clause on a
compression-only ALTER as having nothing to wait on, even though such an
ALTER transitions through a hydrate-overlap like any other shape change.
Allow it.

The observability surface had it too: `mz_cluster_reconfigurations.changes`
diffed only the other four dimensions, so a compression-only record
reported an empty diff while in progress, and the SHOW CLUSTERS activity
summary showed such a cluster as steady. Both now report the dimension.

The compression ALTERs in managed_cluster.slt relied on the realized-config
leak: their SHOW CREATE readback only saw the new value because the reshape
wrote it early. They now request the synchronous cut-over (WAIT FOR '0s'),
which advances the realized config in the ALTER's own transaction. ALTER
... RESET takes no WITH clause, so the RESET spelling is exercised for
acceptance without a readback. cluster-controller.td gains regression
coverage: the cc_preserve section asserts that a folding compression ALTER
lands in the record's target and changes while the realized config stays
untouched and only the overlap replica is bounced, and a new section runs a
compression-only ALTER through the record path end to end.

Changing the MV's definition changes its builtin fingerprint, so the
builtin schema migration registers a replacement step for
`mz_cluster_reconfigurations`, and the catalog-docs golden and the
mz_show_clusters plan golden are rewritten.
@aljoscha
aljoscha force-pushed the aljoscha/cluster-legacy-06-compression-fix branch from cf73bc0 to 8f11dd9 Compare August 7, 2026 12:28
@aljoscha
aljoscha force-pushed the aljoscha/cluster-legacy-04-zdt-flag branch from c6c719e to d9f10e3 Compare August 10, 2026 17:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant